{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1C. Data Cleaning: Zipcode Clusterings\n",
    "<hr>\n",
    "\n",
    "We take the same approach to cleaning the data as in our original version. We take zipcodes and cluster them into neighborhoods. This is better than using the original neighborhoods column as there were many missing values and non-uniform neighborhood names. By converting by cleaned zipcode, we can ensure that the clustering by neighborhoods is more accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import seaborn as sb\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cmx\n",
    "import matplotlib.colors as colors\n",
    "import math\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols = [\n",
    "    'id',\n",
    "    'host_id',\n",
    "    'zipcode',\n",
    "    'property_type',\n",
    "    'room_type',\n",
    "    'accommodates',\n",
    "    'bedrooms',\n",
    "    'beds',\n",
    "    'bed_type',\n",
    "    'price',\n",
    "    'number_of_reviews',\n",
    "    'review_scores_rating',\n",
    "    'host_listing_count',\n",
    "    'availability_30',\n",
    "    'minimum_nights',\n",
    "    'bathrooms'\n",
    "]\n",
    "\n",
    "data = pd.read_csv('../datasets/raw_datasets/listings.csv', usecols=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove NaN values from dataframe\n",
    "data = data.dropna(how='any', subset=['zipcode', 'property_type', 'bedrooms', 'beds', 'bathrooms'])\n",
    "\n",
    "# convert formatting for price\n",
    "data['price'] = (data['price'].str.replace(r'[^-+\\d.]', '').astype(float))\n",
    "\n",
    "# drop any inconsistent values\n",
    "data = data[data['accommodates'] != 0]\n",
    "data = data[data['bedrooms'] != 0]\n",
    "data = data[data['beds'] != 0]\n",
    "data = data[data['price'] != 0.00]\n",
    "\n",
    "# convert ZipCode\n",
    "data['zipcode'] = data['zipcode'].str.replace(r'-\\d+', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# turn NaN scores with 0 reviews into 'No Reviews'\n",
    "idx_vals = data['review_scores_rating'][data['number_of_reviews'] == 0].index.values.tolist()\n",
    "data.loc[idx_vals, ('review_scores_rating')] = data['review_scores_rating'][data['number_of_reviews'] == 0].replace('NaN', 'No Reviews')\n",
    "\n",
    "# remove inconsistent NaN values\n",
    "data = data[~data['review_scores_rating'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# convert review_scores_rating into buckets\n",
    "def convert_scores_buckets(val):\n",
    "    if val == 'No Reviews':\n",
    "        return 'No Reviews'\n",
    "    elif val >= 95.0:\n",
    "        return '95-100'\n",
    "    elif val >= 90.0 and val < 95.0:\n",
    "        return '90-94'\n",
    "    elif val >= 85.0 and val < 90.0:\n",
    "        return '85-89'\n",
    "    elif val >= 80.0 and val < 85.0:\n",
    "        return '80-84'\n",
    "    elif val >= 70.0 and val < 80.0:\n",
    "        return '70-79'\n",
    "    elif val >= 60.0 and val < 70.0:\n",
    "        return '60-69'\n",
    "    elif val >= 50.0 and val < 60.0:\n",
    "        return '50-59'\n",
    "    elif val >= 40.0 and val < 50.0:\n",
    "        return '40-49'\n",
    "    elif val >= 30.0 and val < 40.0:\n",
    "        return '30-39'\n",
    "    elif val >= 20.0 and val < 30.0:\n",
    "        return '20-29'\n",
    "    elif val >= 10.0 and val < 20.0:\n",
    "        return '10-19'\n",
    "    elif val < 10.0:\n",
    "        return '0-9'\n",
    "    \n",
    "data['review_scores_rating'] = data['review_scores_rating'].apply(convert_scores_buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now will cluster the zipcodes together based on their neighborhood as described at the following link: https://www.health.ny.gov/statistics/cancer/registry/appendix/neighborhoods.htm\n",
    "\n",
    "We also encoded zip codes not listed here by placing them with their neighbors (11249). We do this clustering because the neighborhood might have more important information than a zipcode. People would be more likely to search a specific neighborhood on air BN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# convert zipcodes into neighbrohoods\n",
    "\n",
    "dict = {\n",
    "    # Bronx Neighborhoods\n",
    "    '10453': 'Central Bronx',\n",
    "    '10457': 'Central Bronx',\n",
    "    '10460': 'Central Bronx',\n",
    "    '10458': 'Bronx Park and Fordham',\n",
    "    '10467': 'Bronx Park and Fordham',\n",
    "    '10468': 'Bronx Park and Fordham',\n",
    "    '10451': 'High Bridge and Morrisania',\n",
    "    '10452': 'High Bridge and Morrisania',\n",
    "    '10456': 'High Bridge and Morrisania',\n",
    "    '10454': 'Hunts Point and Mott Haven',\n",
    "    '10455': 'Hunts Point and Mott Haven',\n",
    "    '10459': 'Hunts Point and Mott Haven',\n",
    "    '10474': 'Hunts Point and Mott Haven',\n",
    "    '10045': 'Hunts Point and Mott Haven',\n",
    "    '10463': 'Kingsbridge and Riverdale',\n",
    "    '10471': 'Kingsbridge and Riverdale',\n",
    "    '10466': 'Northeast Bronx',\n",
    "    '10469': 'Northeast Bronx',\n",
    "    '10470': 'Northeast Bronx',\n",
    "    '10475': 'Northeast Bronx',\n",
    "    '10704': 'Northeast Bronx',\n",
    "    '10461': 'Southeast Bronx',\n",
    "    '10462': 'Southeast Bronx',\n",
    "    '10464': 'Southeast Bronx',\n",
    "    '10465': 'Southeast Bronx',\n",
    "    '10472': 'Southeast Bronx',\n",
    "    '10473': 'Southeast Bronx',\n",
    "    # Brooklyn Neighborhoods\n",
    "    '11212': 'Central Brooklyn',\n",
    "    '11213': 'Central Brooklyn',\n",
    "    '11216': 'Central Brooklyn',\n",
    "    '11233': 'Central Brooklyn',\n",
    "    '11238': 'Central Brooklyn',\n",
    "    '11209': 'Southwest Brooklyn',\n",
    "    '11214': 'Southwest Brooklyn',\n",
    "    '11228': 'Southwest Brooklyn',\n",
    "    '11204': 'Borough Park',\n",
    "    '11218': 'Borough Park',\n",
    "    '11219': 'Borough Park',\n",
    "    '11230': 'Borough Park',\n",
    "    '11234': 'Canarsie and Flatlands',\n",
    "    '11236': 'Canarsie and Flatlands',\n",
    "    '11239': 'Canarsie and Flatlands',\n",
    "    '11223': 'Southern Brooklyn',\n",
    "    '11224': 'Southern Brooklyn',\n",
    "    '11229': 'Southern Brooklyn',\n",
    "    '11235': 'Southern Brooklyn',\n",
    "    '11201': 'Northwest Brooklyn',\n",
    "    '11205': 'Northwest Brooklyn',\n",
    "    '11215': 'Northwest Brooklyn',\n",
    "    '11217': 'Northwest Brooklyn',\n",
    "    '11231': 'Northwest Brooklyn',\n",
    "    '11203': 'Flatbush',\n",
    "    '11210': 'Flatbush',\n",
    "    '11225': 'Flatbush',\n",
    "    '11226': 'Flatbush',\n",
    "    '11126': 'Flatbush',\n",
    "    '11207': 'East New York and New Lots',\n",
    "    '11208': 'East New York and New Lots',\n",
    "    '11211': 'Greenpoint',\n",
    "    '11222': 'Greenpoint',\n",
    "    '11220': 'Sunset Park',\n",
    "    '11232': 'Sunset Park',\n",
    "    '11206': 'Bushwick and Williamsburg',\n",
    "    '11221': 'Bushwick and Williamsburg',\n",
    "    '11237': 'Bushwick and Williamsburg',\n",
    "    '11249': 'Bushwick and Williamsburg',\n",
    "    # Staten Island Neighborhoods\n",
    "    '10302': 'Port Richmond',\n",
    "    '10303': 'Port Richmond',\n",
    "    '10310': 'Port Richmond',\n",
    "    '10306': 'South Shore',\n",
    "    '10307': 'South Shore',\n",
    "    '10308': 'South Shore',\n",
    "    '10309': 'South Shore',\n",
    "    '10312': 'South Shore',\n",
    "    '10301': 'Stapleton and St. George',\n",
    "    '10304': 'Stapleton and St. George',\n",
    "    '10305': 'Stapleton and St. George',\n",
    "    '10314': 'Mid-Island',\n",
    "    # Manhattan Neighborhoods\n",
    "    '10026': 'Central Harlem',\n",
    "    '10027': 'Central Harlem',\n",
    "    '10030': 'Central Harlem',\n",
    "    '10037': 'Central Harlem',\n",
    "    '10039': 'Central Harlem',\n",
    "    '10001': 'Chelsea and Clinton',\n",
    "    '10011': 'Chelsea and Clinton',\n",
    "    '10018': 'Chelsea and Clinton',\n",
    "    '10019': 'Chelsea and Clinton',\n",
    "    '10020': 'Chelsea and Clinton',\n",
    "    '10036': 'Chelsea and Clinton',\n",
    "    '1001': 'Chelsea and Clinton',\n",
    "    '10029': 'East Harlem',\n",
    "    '10035': 'East Harlem',\n",
    "    '10010': 'Gramercy Park and Murray Hill',\n",
    "    '10016': 'Gramercy Park and Murray Hill',\n",
    "    '10017': 'Gramercy Park and Murray Hill',\n",
    "    '10022': 'Gramercy Park and Murray Hill',\n",
    "    '10012': 'Greenwich Village and Soho',\n",
    "    '10013': 'Greenwich Village and Soho',\n",
    "    '10014': 'Greenwich Village and Soho',\n",
    "    '10004': 'Lower Manhattan',\n",
    "    '10005': 'Lower Manhattan',\n",
    "    '10006': 'Lower Manhattan',\n",
    "    '10007': 'Lower Manhattan',\n",
    "    '10038': 'Lower Manhattan',\n",
    "    '10280': 'Lower Manhattan',\n",
    "    '10282': 'Lower Manhattan',\n",
    "    '10080': 'Lower Manhattan',\n",
    "    '10281': 'Lower Manhattan',\n",
    "    '10002': 'Lower East Side',\n",
    "    '10003': 'Lower East Side',\n",
    "    '10009': 'Lower East Side',\n",
    "    '10021': 'Upper East Side',\n",
    "    '10028': 'Upper East Side',\n",
    "    '10044': 'Upper East Side',\n",
    "    '10065': 'Upper East Side',\n",
    "    '10075': 'Upper East Side',\n",
    "    '10128': 'Upper East Side',\n",
    "    '10162': 'Upper East Side',\n",
    "    '8456422473 call for more details': 'Upper East Side',\n",
    "    '10023': 'Upper West Side',\n",
    "    '10024': 'Upper West Side',\n",
    "    '10025': 'Upper West Side',\n",
    "    '10069': 'Upper West Side',\n",
    "    '14072': 'Upper West Side',\n",
    "    '10031': 'Inwood and Washington Heights',\n",
    "    '10032': 'Inwood and Washington Heights',\n",
    "    '10033': 'Inwood and Washington Heights',\n",
    "    '10034': 'Inwood and Washington Heights',\n",
    "    '10040': 'Inwood and Washington Heights',\n",
    "    # Queens Neighborhoods\n",
    "    '11361': 'Northeast Queens',\n",
    "    '11362': 'Northeast Queens',\n",
    "    '11363': 'Northeast Queens',\n",
    "    '11364': 'Northeast Queens',\n",
    "    '11354': 'North Queens',\n",
    "    '11355': 'North Queens',\n",
    "    '11356': 'North Queens',\n",
    "    '11357': 'North Queens',\n",
    "    '11358': 'North Queens',\n",
    "    '11359': 'North Queens',\n",
    "    '11360': 'North Queens',\n",
    "    '11365': 'Central Queens',\n",
    "    '11366': 'Central Queens',\n",
    "    '11367': 'Central Queens',\n",
    "    '11412': 'Jamaica',\n",
    "    '11423': 'Jamaica',\n",
    "    '11432': 'Jamaica',\n",
    "    '11433': 'Jamaica',\n",
    "    '11434': 'Jamaica',\n",
    "    '11435': 'Jamaica',\n",
    "    '11436': 'Jamaica',\n",
    "    '11101': 'Northwest Queens',\n",
    "    '11102': 'Northwest Queens',\n",
    "    '11103': 'Northwest Queens',\n",
    "    '11104': 'Northwest Queens',\n",
    "    '11105': 'Northwest Queens',\n",
    "    '11106': 'Northwest Queens',\n",
    "    '111006': 'Northwest Queens',\n",
    "    '11109': 'Northwest Queens',\n",
    "    '11374': 'West Central Queens',\n",
    "    '11375': 'West Central Queens',\n",
    "    '11379': 'West Central Queens',\n",
    "    '11385': 'West Central Queens',\n",
    "    '11691': 'Rockaways',\n",
    "    '11692': 'Rockaways',\n",
    "    '11693': 'Rockaways',\n",
    "    '11694': 'Rockaways',\n",
    "    '11695': 'Rockaways',\n",
    "    '11697': 'Rockaways',\n",
    "    '11004': 'Southeast Queens',\n",
    "    '11005': 'Southeast Queens',\n",
    "    '11411': 'Southeast Queens',\n",
    "    '11413': 'Southeast Queens',\n",
    "    '11422': 'Southeast Queens',\n",
    "    '11426': 'Southeast Queens',\n",
    "    '11427': 'Southeast Queens',\n",
    "    '11428': 'Southeast Queens',\n",
    "    '11429': 'Southeast Queens',\n",
    "    '11414': 'Southwest Queens',\n",
    "    '11415': 'Southwest Queens',\n",
    "    '11416': 'Southwest Queens',\n",
    "    '11417': 'Southwest Queens',\n",
    "    '11418': 'Southwest Queens',\n",
    "    '11419': 'Southwest Queens',\n",
    "    '11420': 'Southwest Queens',\n",
    "    '11421': 'Southwest Queens',\n",
    "    '11368': 'West Queens',\n",
    "    '11369': 'West Queens',\n",
    "    '11370': 'West Queens',\n",
    "    '11372': 'West Queens',\n",
    "    '11373': 'West Queens',\n",
    "    '11377': 'West Queens',\n",
    "    '11378': 'West Queens',\n",
    "}\n",
    "\n",
    "def convert_zips_hoods(val):\n",
    "     return dict[val]\n",
    "\n",
    "data['neighborhood'] = data['zipcode'].apply(convert_zips_hoods)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = data.drop('zipcode', 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# encode categorical variables\n",
    "neighborhood_dummies = pd.get_dummies(data['neighborhood'])\n",
    "property_dummies = pd.get_dummies(data['property_type'])\n",
    "room_dummies = pd.get_dummies(data['room_type'])\n",
    "bed_dummies = pd.get_dummies(data['bed_type'])\n",
    "ratings_scores_dummies = pd.get_dummies(data['review_scores_rating'])\n",
    "\n",
    "# replace the old columns with our new one-hot encoded ones\n",
    "df = pd.concat((data.drop(['neighborhood', \\\n",
    "    'property_type', 'room_type', 'bed_type', 'review_scores_rating'], axis=1), \\\n",
    "    neighborhood_dummies.astype(str), property_dummies.astype(int), \\\n",
    "    room_dummies.astype(int), bed_dummies.astype(int), ratings_scores_dummies.astype(int)), \\\n",
    "    axis=1)\n",
    "\n",
    "# move target predictor 'price' to the end of the dataframe\n",
    "cols = list(df.columns.values)\n",
    "idx = cols.index('price')\n",
    "rearrange_cols = cols[:idx] + cols[idx+1:] + [cols[idx]]\n",
    "df = df[rearrange_cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>host_id</th>\n",
       "      <th>accommodates</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>beds</th>\n",
       "      <th>minimum_nights</th>\n",
       "      <th>availability_30</th>\n",
       "      <th>number_of_reviews</th>\n",
       "      <th>host_listing_count</th>\n",
       "      <th>...</th>\n",
       "      <th>40-49</th>\n",
       "      <th>50-59</th>\n",
       "      <th>60-69</th>\n",
       "      <th>70-79</th>\n",
       "      <th>80-84</th>\n",
       "      <th>85-89</th>\n",
       "      <th>90-94</th>\n",
       "      <th>95-100</th>\n",
       "      <th>No Reviews</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1069266</td>\n",
       "      <td>5867023</td>\n",
       "      <td>-0.520256</td>\n",
       "      <td>-0.331519</td>\n",
       "      <td>-0.407440</td>\n",
       "      <td>-0.493059</td>\n",
       "      <td>0.173446</td>\n",
       "      <td>0.390321</td>\n",
       "      <td>2.716276</td>\n",
       "      <td>-0.355986</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>160.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2061725</td>\n",
       "      <td>4601412</td>\n",
       "      <td>-0.520256</td>\n",
       "      <td>-0.331519</td>\n",
       "      <td>-0.407440</td>\n",
       "      <td>0.381668</td>\n",
       "      <td>0.173446</td>\n",
       "      <td>-0.965980</td>\n",
       "      <td>1.295702</td>\n",
       "      <td>0.932775</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>58.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>44974</td>\n",
       "      <td>198425</td>\n",
       "      <td>-0.520256</td>\n",
       "      <td>-0.331519</td>\n",
       "      <td>-0.407440</td>\n",
       "      <td>-0.493059</td>\n",
       "      <td>2.885991</td>\n",
       "      <td>-1.205327</td>\n",
       "      <td>0.822177</td>\n",
       "      <td>-0.355986</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>185.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4701675</td>\n",
       "      <td>22590025</td>\n",
       "      <td>-0.520256</td>\n",
       "      <td>-0.331519</td>\n",
       "      <td>-0.407440</td>\n",
       "      <td>0.381668</td>\n",
       "      <td>-0.601567</td>\n",
       "      <td>1.108363</td>\n",
       "      <td>-0.493170</td>\n",
       "      <td>-0.355986</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>195.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>68914</td>\n",
       "      <td>343302</td>\n",
       "      <td>1.690843</td>\n",
       "      <td>-0.331519</td>\n",
       "      <td>1.266082</td>\n",
       "      <td>1.256396</td>\n",
       "      <td>-0.214061</td>\n",
       "      <td>-0.407503</td>\n",
       "      <td>0.296038</td>\n",
       "      <td>0.073601</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>165.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 90 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id   host_id  accommodates  bathrooms  bedrooms      beds  \\\n",
       "0  1069266   5867023     -0.520256  -0.331519 -0.407440 -0.493059   \n",
       "2  2061725   4601412     -0.520256  -0.331519 -0.407440  0.381668   \n",
       "3    44974    198425     -0.520256  -0.331519 -0.407440 -0.493059   \n",
       "4  4701675  22590025     -0.520256  -0.331519 -0.407440  0.381668   \n",
       "5    68914    343302      1.690843  -0.331519  1.266082  1.256396   \n",
       "\n",
       "   minimum_nights  availability_30  number_of_reviews  host_listing_count  \\\n",
       "0        0.173446         0.390321           2.716276           -0.355986   \n",
       "2        0.173446        -0.965980           1.295702            0.932775   \n",
       "3        2.885991        -1.205327           0.822177           -0.355986   \n",
       "4       -0.601567         1.108363          -0.493170           -0.355986   \n",
       "5       -0.214061        -0.407503           0.296038            0.073601   \n",
       "\n",
       "   ...   40-49 50-59 60-69 70-79 80-84 85-89 90-94 95-100 No Reviews  price  \n",
       "0  ...       0     0     0     0     0     1     0      0          0  160.0  \n",
       "2  ...       0     0     0     0     0     0     0      1          0   58.0  \n",
       "3  ...       0     0     0     0     0     0     0      1          0  185.0  \n",
       "4  ...       0     0     0     0     0     0     0      1          0  195.0  \n",
       "5  ...       0     0     0     0     0     0     0      1          0  165.0  \n",
       "\n",
       "[5 rows x 90 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert non-categorical variables to floats and standardize\n",
    "def standardize_col(col):\n",
    "    mean = np.mean(col)\n",
    "    std = np.std(col)\n",
    "    return col.apply(lambda x: (x - mean) / std)\n",
    "\n",
    "non_cat_vars = ['accommodates', 'bedrooms', 'beds', 'number_of_reviews', 'host_listing_count', 'availability_30', 'minimum_nights', 'bathrooms']\n",
    "for col in non_cat_vars:\n",
    "    df[col] = df[col].astype(float)\n",
    "    df[col] = standardize_col(df[col])\n",
    "    \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# log transform the response 'price'\n",
    "df['price_log'] = df['price'].apply(lambda x: math.log(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note:\n",
    "\n",
    "This data set is different from the original because it now has neighborhood booleans as opposed to zipcode booleans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read to csv\n",
    "df.to_csv('../datasets/listings_neighborhood_clean.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
